Evaluating the Consistency of DETECT Indices and Item Clusters Using Simulated and Real Data that Display both Simple and Complex Structure

DETECT is a nonparametric procedure based on conditional covariances for dimensionality assessment. It can be used to assess the strength of multidimensionality in a test, to estimate the number of dimensions, and to partition items into distinct clusters that represent dimensions underlying the test. DETECT works well with test data that display simple or approximate simple structure in simulated and real data studies (e.g., Roussos & Ozbek, 2005; Zhang & Stout, 1999b). However, its performance with data that display complex structure has only been systematically evaluated in one published study (Gierl, Leighton, & Tan, in press). The present study evaluates the performance of DETECT under conditions of both approximate simple and complex structures using simulated data. The connection between the simulated outcomes and real testing situations is also studied using data from the SAT 2005 March administration. The results from the simulation studies suggest that DETECT can adequately identify the dimensional structure of tests (with 80% or higher classification accuracy and consistency) for most of the approximate simple structure conditions (except when items with lower discrimination powers are involved and the correlation is high [ 0.80]) and for some complex structure conditions. Degree of complexity, correlation between dimensions, and item discrimination power all affect the DETECT results. A relaxed evaluation criterion for is proposed, and a recommendation to interpret the index relative to three independent variables—degree of complexity, correlation between dimensions, and item discrimination power—is also proposed in the present study. ≥